CathAI: fully automated coronary angiography interpretation and stenosis estimation

Coronary angiography is the primary procedure for diagnosis and management decisions in coronary artery disease (CAD), but ad-hoc visual assessment of angiograms has high variability. Here we report a fully automated approach to interpret angiographic coronary artery stenosis from standard coronary angiograms. Using 13,843 angiographic studies from 11,972 adult patients at University of California, San Francisco (UCSF), between April 1, 2008 and December 31, 2019, we train neural networks to accomplish four sequential necessary tasks for automatic coronary artery stenosis localization and estimation. Algorithms are internally validated against criterion-standard labels for each task in hold-out test datasets. Algorithms are then externally validated in real-world angiograms from the University of Ottawa Heart Institute (UOHI) and also retrained using quantitative coronary angiography (QCA) data from the Montreal Heart Institute (MHI) core lab. The CathAI system achieves state-of-the-art performance across all tasks on unselected, real-world angiograms. Positive predictive value, sensitivity and F1 score are all ≥90% to identify projection angle and ≥93% for left/right coronary artery angiogram detection. To predict obstructive CAD stenosis (≥70%), CathAI exhibits an AUC of 0.862 (95% CI: 0.843–0.880). In UOHI external validation, CathAI achieves AUC 0.869 (95% CI: 0.830–0.907) to predict obstructive CAD. In the MHI QCA dataset, CathAI achieves an AUC of 0.775 (95%. CI: 0.594–0.955) after retraining. In conclusion, multiple purpose-built neural networks can function in sequence to accomplish automated analysis of real-world angiograms, which could increase standardization and reproducibility in angiographic coronary stenosis assessment.

We examined the characteristics of those patients who were determined to have obstructive AI-stenosis (</³70%) that were either concordant (1,336) or discordant (398) with the REPORT-stenosis (Extended Figure 9). We specifically refrain from using the terms "false positive/false negative" in this setting, since as we discussed, the REPORT-stenosis is subject to error and non-trivial variability. AI-stenosis was more likely to be discordant with REPORTstenosis in older patients (62.7±13.2 vs 65.1±12.3, <0.001), in the LCA, the proximal RCA, distal RCA, the right posterolateral and the distal LAD.
To examine the performance of our SSIM-based approach to identify the peak-contrast frame, we examined the ratio of the peak-contrast frame number over the total frames in a video.
Since contrast dye injection usually begins within the first 0.5 seconds of the typical 4-6 second angiogram video, we would expect most videos to exhibit their peak-contrast frame within the first 20-50% (ratio 0.2-0.5) of frames. Across the Full Dataset, the SSIM-selected peak-contrast frame ratio was 0.33±0.18 (Extended Figure 10), which is consistent with expectation.
The current state-of-the-art for assisted coronary stenosis assessment primarily encompasses QCA 1 , which still relies upon significant human input and exhibits significant variability. A study assessing 10 different QCA systems against a phantom stenosis goldstandard found absolute percentage differences of -26% to +29% in coronary stenosis assessments between systems 2 . For different individuals using the same QCA system an 11.2% absolute percentage difference was reported. Notably, on our post-hoc review of examples of angiograms where AI-stenosis and REPORT-stenosis were discordant, the REPORT-stenosis value was not always objectively more correct. Because the mean absolute difference between AI-stenosis and REPORT-stenosis of ~18% was so similar to human inter-observer variability 3,4 (which is also likely present in the clinically-generated Report dataset), CathAI performance may be substantially improved if re-trained using less variable training labelssuch as with core lab-generated QCA estimates. Therefore, QCA or other stenosis assessment assistance methods could play an important role to provide large numbers of high-quality training labels with which to further improve CathAI. Similarly, training dedicated algorithms for certain higher-volume angiographic projections, like our Algorithm 3B, could decrease input variability into all models and improve overall performance.
The model explainability methods we performed highlight how individual algorithms in CathAI function to accomplish their tasks. GradCAM and LOVI suggest that the algorithms often focused on similar regions of the image as a human expert does to classify anatomy and predict stenosis severity. For stenosis assessment (Algorithm 4), LOVI showed that CathAI involved pixels not only at the narrowest part of the stenotic artery but also outside of the area of stenosis, like how human cardiologists use both normal and abnormal segments of the coronary artery to assess relative severity.
CathAI's could also be trained with additional labels to define overall CHD burden, like an automated SYNTAX score 5 , to guide revascularization decisions. The CathAI pipeline provides a foundation upon which a wide range of clinically relevant applications related to automated angiographic analysis can be trained with different task-specific labels. For example, CathAI could be trained to predict FFR or IFR values directly from angiographic videos containing stenosis or to highlight poorly-visible objects like prior stents, collateral arteries or bypass graft sites.
Additional limitations include that the text-parsing method we used to extract the REPORT-stenosis from the clinical procedure report may have introduced errors in either the location of the stenosis or the degree of severity. Resultant variability in stenosis labels used for training, either from clinical variability or parsing, would be expected to adversely impact algorithm performance by biasing results to the null and decreasing the observed effect of association. Perhaps in part due to this variability in the stenosis training labels, our algorithm replicated the clinical bias and tended to underestimate severe stenoses and overestimate minor stenoses. This may also have been impacted, in part, by having fewer training examples of severe stenosis.

Supplementary Figures
Supplementary Figure 1

Coronary segment
Proximal RCA* From ostium to one half the distance to the acute margin of the heart.
Middle RCA* From end of first segment to acute margin of heart.
Distal RCA* From the acute margin of the heart to the origin of the posterior descending artery.
Posterior descending artery* Artery running the posterior interventricular groove.